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(54) Method and apparatus for providing plug-in media decoders 

(57) A method and apparatus for providing plug-in 
media decoders. Embodiments provide a "plug-in" 
decoder architecture that allows software decoders to 
be transparently downloaded, along with media data. 
User applications are able to support new media types 
as long as the corresponding plug-in decode* is availa- 
ble with the media data. Persistent storage require- 
ments are decreased because the downloaded decoder 
is transient, existing in application memory for the dura- 
tion of execution of the user application. The architec- 
ture also supports use of plug-in decoders already 
installed in the user computer. One embodiment is 
implemented with object-based class files executed in a 
virtual machine to form a media application. A media 
data type is determined from incoming media data, and 
used to generate a class name for a corresponding 
codec (coder-decoder) object. A class path vector is 
searched, including the source location of the incoming 
media data, to determine the location of the codec class 
file for the given class name. When the desired codec 
class file is located, the virtual machine's class loader 
loads the class file for integration into the media applica- 
tion. If the codec class file is located across the network 
at the source location of the media data, the class 
loader downloads the codec class file from the network. 
Once the class file is loaded into the virtual machine, an 
instance of the codec class is created within the media 
application to decode/decompress the media data as 
appropriate for the media data type. 
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Description 

RAP.KfiROMND OF T HE INVENTION 
5 1 FIELD OF THE INVENTION 

[0001] This invention relates to the field of object-oriented computer applications, and. more specifically, to network 
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SSSTm** «* * e disc,osure of this P atent document contain is subJ f 10 "2 9 £ P TZ'^ 

Uyr 9ht owner has no objection to the facsimile reproduction by anyone of the patent d ^ um «^^"L ^ 
sureas it appears in the Patent and Trademark Office file or records, but otherw.se reserves all copynght rights what- 
soever. 

2. gAQSaRQUMLABI 

[0003] Growing interest in the media distribution capabilities of the Internet and the World Wide Web has resulted h 
the development of a variety of multimedia data compression and encoding formats or media types, such as MPEG-i . 
KSrl, H.26?, PCM. AU and MIDI, for efficient transport of audio, video and other forms d 
applications supporting one or more such media types are available for presenting murt.med .a content d ^loaded 
from the Internet or other networks. To enhance the capabilities of those media appl.cat.ons. plug-in decoders are often 
installed on the user machine to facilitate decompression/decoding of particular media types resources 
[0004] Unfortunately, the installation of plug-in decoders becomes cumbersome in terms of storage resources 
Z£»d for Sling the plug-ins on the user computer system. Also, a user is required to locate the often e usive plug- 
nTe^eTaS install them on the user computer system prior to viewing any media data of a 9-^ ^rmat The uWrty 
of a media application is thus limited to those media formats supported by a user's coUecton of 
ers. With thelarge numbers of current medfc types, and the constant influx of new media types, rt .snot feasrt* rto nM 
every type of plug-in decoder. To better illustrate the shortcomings of current media applications, descnption of a sam- 
ple networked multimedia environment of the prior art is given below with respectto ^Rgure T 
[0005] in Figure 1 . user computer system 100 is coupled to server 106 "a network ^^^^^TSSSL 
sistent storage 107. which contains a library of media data of various data types (e.g.. JPEG, PCM. MPEG1 . OnepaK. 
SETrne media data include data files 108. 109 and 1 10 of data types JPEG. PCM and MPEG1 , respectively. 
SooeT UsTconpuU system 1 00 comprises a persistent storage resource 1 01 . such as a hard drive and an execu- 
tonSace I0 3 7n temporary storage such as system RAM. Persistent storage 101 contains a plugnn library 102. «m- 
oSr^nstanedplug^^ 112 and 113 corresponding to JPEG. PCM and H.261 date types -""Pf"* 

SStaipm 1W contains media application 104. which comprises JPEG datefile 108' and plug-m JPEG decoder 

111' for decoding/decompressing JPEG media data. ■ . . . . . . 

00071 JPEG date file 108' is a copy of data fie 108. downloaded from server 1 06. Plug-in decoder 1 1 1 is a copy of 
J^G decoder 1 iTloaded from persistent storage 101 to accommodate processing of JPEG date. Because user com- 
putefsysS ^ Ooonvnasplug-Sndecodersfor JPEG. PCM and H.261 datatypes instated in P«««^^"; 
media aoolication 104 is limited to presenting either JPEG, PCM or H.261 media data. For example. MPEG1 data file 

sistem storage 101 for decoding/decompressing MPEG1 media data. There is no mechanism for decoding media ^ 
of data types for which a corresponding plug-in is not installed on the user computer system. Further, unused plug-in 
decoders installed on user computer system 100 represent an ineff icient use of storage resources. 
[00081 For systems such as diskless clients and Network Computers (NCs). which are not equipped with persistent 
storaqe such as a hard disk, pre-installation of media decoders is not supported. A plug-in architecture may -not be pos- 
SwSng prientation of media to a small number of data types supported by decoders built into the media appli- 
cation itself. No flexibility is provided for new data types. 

so SI IMMARY OF THF INVENTION 

[0009] AmethodarxlapparatusforprovM^^ !^T^2^|^3E 
vide a "plug-in" decoder architecture that allows software decoders to be transparently downloaded, along with media 
data, from a network to a user application on a user computer. The user application is thus 
55 types as long as the corresponding plug-in decoder is available over the network with the media data. P^ent stor- 
age requirements are lessened because the downloaded decoder is transient, existing in application memory for the 
duratiS of execution of the user application. The architecture also supports use of plug-in decoders already installed 
in the user computer. 
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[0010] One embodiment of the invention is implemented with object-based class files executed m a virtual machine 
to form a media application. A media data type is determined from incoming media data, and used tog^rateada^ 
name for a codecs-decoder) object corresponding to the giver, media data type^A search ■ - 
lions orovided in a class path vector, including the source location of the incom.ng media data, to determine flie location 
tfft. Zt f i eSr the given class name. When the desired codec class file is located, the virtual machine's dass 
loader^S 7he1.ass fie for integration into the media application. If the codec Cass file is located ac^e n^work 
auhe source location of the media data, the class loader downloads the codec class file from the network. Once the 
class file is loaded into the virtual machine, an instance of the codec class is created within the media application to 
decode/decompress the media data as appropriate for the media data type. 



dq F p pFRP.RIPTION OF THF DRAWINGS 
[0011] 

is Figure 1 is a block diagram of a networked multimedia system of the prior art. 

Figure 2 is a block diagram of a networked multimedia system in accordance with an embodiment of the invention. 

Figure 3 is a block diagram of a media processing pipeline in a media application. 

20 Figure 4 is a flow diagram of a method for providing a plug-in media decoder in accordance with an embodiment of 
the invention. 

Figure 5 is a block diagram of software object apparatus for creating a codec instance in accordance with an 
25 embodiment of the invention. 

Figure 6 is a block diagram of an embodiment of a computer system capable of providing a suitable execution envi- 
ronment for an embodiment of the invention. 

30 DETAILED DESCRIPTION Q F THF INVENTION 

[0012] The invention is a method and apparatus for providing plug-in media decoders. In the following desc ri P« on ; 
nume ousjecif ic details are set forth to provide a more thorough description of embodiments of the i^ertioa tt wd 
be^PparenThowever. to one skilledinthe art. that the invention may be practiced without these^>eaf.c details, mother 
instances, well known features have not been described in detail so as not to obscure the '^nton. 
[0013] A-plug-in-decoderarchrtectureisprovidedwhichallo^softwaredec^^ 

along with media data, and integrated into a media application for use in processing the media data The media appu- 
cat^Siuslble to support new media types as long as the corresponding plug-in decoder is available over the in* 
w^r^h me media daTrhe downloading process and the integration of the decoder into the user JWl^on * 
transparent to the user. Further, the downloaded decoder is transient, existing in application m ««^"^^£ 
execution of the user application. Therefore, persistent storage resources, such as space <^^£*££. 
required for permanently installing plug-in decoders. Diskless clients and Network Computers (NCs) are therefore able 
to take advantage of the flexibility provided by the plug-in decoder architecture. 

IOOU] The plug-in decoder architecture also supports plug-in decoders that are not downloaded, but are locally 
Sled (stored in persistent storage) on the user computer system. The installed plug-in decoder can thus be used. 

when present, to minimize download time. .... _ mo Hia 

[001 5] Figure 2 is a block diagram of a network environment illustrating the provision of plug-in decoders wrth medra 
date in accordance with an embodiment of the invention. A server is provided thatcontains media 
types, as well as corresponding decoders tor use in decoding/decompressing each type of media data P^^ A user 
computer system contains a media application for presenting media data. A limited number of plug-m decoders are pro- 
vided in the user computer system. _ em ^ ine(vmwitten(ir . 
[0016] in Figure 2. user computer system 100 is coupled to server 106 via 

sistenJ storage 1 07. which contains a library of media data of various types (eg JPEG. PCM. HgM JKPEGJ ■ MPS* 
Cinepak. MIDI. etc.). The media data include data files 108. 109 and 110 of datatypes Jp EG. PCMandH^1.r^ec 
55 ^Corresponding plug-in decoders for each data type are also stored within persistent storage 107. including 
decoded 1 1? 112 and 1 14. for decoding/decompressing JPEG. PCM and H.261 medjadata. .y** 
[<£l7] User computer system 100 comprises a persistent storage 101 . such as a hard dnve. and an mcuhom *«ce 
103 in temporary Storage such as system RAM. Persistent storage 101 contains a plug-in library 102 comprising 
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installed plug-in decoders 1 12 and 113 corresponding to data types PCM and MPEG1, respectively. Execution space 
103 contains media application 104, which comprises JPEG data file 108' and PCM data file 109\ Application 104 also 
comprises plug-in decoders 1 1 V and 1 12' for decoding/decompressing JPEG and PCM media data, respectively. 
[0018] Data files 108' and 109' are transient copies of data files 108 and 109 downloaded from server 106. Similarly, 
plug-in JPEG decoder 1 1 1 ' is a transient copy of plug-in JPEG decoder 1 1 1 downloaded from server 1 06 to accommo- 
date processing of JPEG media data. Plug-in PCM decoder 112' is a copy of plug-in PCM decoder 112 loaded from 
persistent storage 101 . Alternatively, because plug-in decoder 1 12 for media data of type PCM is both installed on user 
computer system 100 and on server 106. plug-in PCM decoder 112' may be downloaded from server 106. When appli- 
cation 1 04 shuts down, plug-in decoders 1 1 1 ' and 1 1 2' are deleted from execution space 1 03. 
[0019] Because user corrputer system 100 only has plug-in decoders for data types PCM and MPEG1 installed in 
persistent storage 101 . under prior art systems, media application 104 would be limited to presenting media of data 
types PCM or MPEG1 . However, because plug-in decoders are provided from the server for each type of data available 
from the server in the form of data files, and because a mechanism is provided for downloading the plug-in decoders 
from the server, the presentation of media data of any of those types (JPEG, PCM, H.261 . etc.) is supported by the sys- 
tem of Figure 2. 

[0020] An embodiment of the nvention utilizes object classes to implement plug-in decoders in an object-oriented 
environment. For a better understanding of object classes, a brief description of object-oriented programming is pro- 
vided below. 

Object-Oriented Programming 

[0021] Object-oriented programming is a method of creating computer programs by combining certain fundamental 
building blocks, and creating relationships among and between the building blocks. The building blocks in object-ori- 
ented programming systems are called 'objects.- An object is a programming unit that groups together a data structure 
(instance variables) and the operations (methods) that can use or affect that data. Thus, an object consists of data and 
one or more operations or procedures that can be performed on that data. The joining of data and operations into a uni- 
tary building block is called "encapsulation." 

[0022] An object can be instructed to perform one of its methods when it receives a "message." A message is a com- 
mand or instruction to the object to execute a certain method. H consists of a method selection (name) and a plurality 
of arguments that are sent to an object. A message tells the receiving object what operations to perform. 
[0023] One advantage of object-oriented programming is the way in which methods are invoked. When a message is 
sent to an object, it is not necessary for the message to instruct the object how to perform a certain method. It is only 
necessary to request that the object execute the method. This greatly simplifies program development. 
[0024] Object-oriented programming languages are predominantly based on a "class" scheme. The class-based 
object-oriented programming scheme is generally described in Lieberman, "Using Prototypical Objects to Implement 
Shared Behavior in Object-Oriented Systems," OOPSLA 86 Proceedings, September 1986, pp. 214-223. 
[0025] A class defines a type of object that typically includes both instance variables and methods for the class. An 
object class is used to create a particular instance of an object. An instance of an object class includes the variables 
and methods defined for the class. Multiple instances of the same class can be created from an object class. Each 
instance that is created from the object class is said to be of the same type or class. 

[0026] A hierarchy of classes can be defined such that an object class definition has one or more subclasses. A sub- 
class inherits its parent's (and grandparent's etc.) definition. Each subclass in the hierarchy may add to or modify the 
behavior specified by its parent class. 

[0027] To illustrate, an employee object class can include "name" and "salary" instance variables and a "set_salary 
method. Instances of the employee object class can be created, or instantiated for each employee in an organization. 
Each object instance is said to be of type "employee." Each employee object instance includes the "name" and "salary" 
instance variables and the "set_salary" method. The values associated with the "name" and "salary" variables in each 
employee object instance contain the name and salary of an employee in the organization. A message can be sent to 
an employee's employee object instance to invoke the "set_salary" method to modify the employee's salary (i.e., the 
value associated with the "salary" variable in the employee's employee object). 

[0028] An object is a generic term that is used in the object-oriented programming environment to refer to a module 
that contains related code and variables. A software application can be written using an object-oriented programming 
language whereby the program's functionality is implemented using objects. 

[0029] Development of software applications may be performed in an independent piecewise manner by establishing 
application programming interfaces (APIs) for components of the application. An API refers to the methods of a partic- 
ular component that are accessible by other components, and the format by which those methods may be invoked. The 
particular implementation of those methods is important only with respect to the design of the particular component. 
Each component is designed individually to implement its respective API and any internal functions, and to interface 
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with the APIs of the other components of the application. Typically, these components comprise one or more objects 

ExS'etTobject-oriented programming languages include C++ and Java®. Unlike most p^ra™g.a^ 
" in which a program is compiled into machine-dependent, executable program code. Ja^dasses are comp.led 
SSSZ independent byte-ccJe class files which are executed by a ^^^^'X^ S Z 
tual machine provides a level of abstraction between the machine independence of the byte-«)de da^esand the 
mach^dependent instruction set of the underlying computer hardware. A class loader ,s re^Ue £52S5l 
byte-code dassf iles as needed, and an interpreter orjust-in-time compiler provides for the transformation of byte-codes 

l^^emSliment of the invention is implemented in the Java programming language as a media ****** con- 
structed us,ng the Java Media Framework (JMF). The media application and the Java Med.a Framework are more fully 
descrfced below. 

pvfln yi^ Multimedia Embodiment of the Invention 

[00321 The Java Med* Framework (JMF) is a set of multimedia APIs and implementations designed to P'^a* mul- 
£SL ■« a variety of protocols and formats, such as a QuickTime Cinepak movie over the HTTP (Hyperte^ Transfer 
P^ocoO or (rtocoTThe Java Media Framework specifies the concept of a "player." a unit to playback multimedia data. 
£££££££ 2 Twithin each JMF player are implementations of the three main stages ^eme^ prcc^ng 
pfpelme the parser 302. the decoder 304 and the renderer 306. Parser 302 transforms the med.a data from protocol 
S n*wo* pa<*et data 301 received from a network source 300 into a compressed and/or encoded media data 
^oder (codec) 304 transforms the compressed and/or encoded media date 303 .^ (decom- 
o essSa^u^ncodea) media data 305 for use by renderer 306. Renderer 306 processes the raw media data 305 
Tr^^a^wS display or presentation on an output device 308. such as a monitor for g-aph.cs or a speaker 

25 SoST A decoder (codec) is a Java object responsible for decoding a single compression format. An example tea 
JPEG decoder which decodes JPEG compressed video frames. A particular implementation of a JMF player may 

include several of these decoder objects for use in viewing multimedia data. u„„ ; ^ m 

TmeWantem » iwrfctod to plus. * '«''>^_ K _^J^^J!^!^^X^^Z 



10 



15 



20 



30 



35 



40 



[0034] A mecnamsm IS proviaea iu piuy in now . " , ^ ^ s^^. rwHor An 

Includes a weiWef ined Java codec object interface, and a search mechan*m to load anc iw^mtmMc. An 
example of a well-defined codec object interface is provided as Appendix A m the form of several related ap is 
S Tan embodiment of the invention, the search mechanism maps well-known codec names to Java , dMB 
nm and n^akesLse of the Java dass loader for the installation of the codec into the JMF player. Tne ^ 
cSoad* provides a flexible approach to install codec classes. Codec classes can h "^*""^£2£J 
variety of sources as specified in the Java classpath. a vector of pathnames to dass packages. This mechan.^ allows 
toSmtfton or wdecdasses that are downloaded from the network, as well as codec classes that are locally and 
inlwSe user computer system. Downloaded codecs have the property ^^^" vst^V 
This means that the downloaded codec classes are downloaded "on the fly" and ex.st on the user conputer system only 

for the duration ot execution of the player. ^*~r*~H in 

[003?] The player can be in one of six states: unrealized, realizing, realized, prefetching, prefetched ar^ started^ In 
nomSl operatton. a player steps through each state until it reaches the "started" state. When a player is fret created, it 
isTn^e^Sed-stL. The player has been instantiated, but does not yet Know anythingabout rts media 
5X5 wTn the realizeO method is called, a player moves from the "unrealized" state into the realizing stateA 
£g pTyer is in the process of determining its resource requirements. ^.^^^^"^ 
resources the player only needs to acquire once. These might include parsing, decoding and 

than exclusive use resources (exdusive-use resources are limited resources such as particular hardware devices that 
can onlv be used by one player at a time; such resources are acquired during "prefetching ). 

" Whin a p'ayer Lshes "realizing." the player moves into the "realized" state. A realized player knows what 
Sees it needs and information about the type of media it is to present. A realized player 
data, and can therefore provide visual components and controls. The player's connections to ^ercbjects .n the sys 
tamara in place but the player does not own any resources that would prevent another player from starting. 

wtn Creech ) method is called, a player moves from the "realized" state ir*, the 
prefetching player is preparing to present its media. During this state, the player preloads rts media data, obtains exdu 
£TeSL and anything else the player needs to do to prepare rtself to play. It is typically during this state that 
55 the olua-in codec is acquired in accordance with an embodiment of the invention. -w^v-h^- 
rooioi Whenlplayer finishes "prefetching." the player moves into the "prefetched" state. Aplayerrnthe Prefetched 
Xs re^ytobesSed. and waits for location of the startO method to transition into the "started" tfate. During the 
"started" state, a player s time-base time and media time are mapped, and the media data is presented. 
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[0041] In accordance with a preferred embodiment of the invention, during the "prefetching" state described above, 
the player obtains a plug-in codec object fa decoding/decompressing the incoming media data. Attentively, the pUig- 
in codec object may be obtained during the "realizing" state. An embodiment of a method by which plug-in decoders 
are located and integrated into a media application is illustrated in the flow diagram of Figure 4. 
[0042] In step 400 the data type of the media data is determined, for example, from the data header information 
extracted by the parser. From the data type information, in step 401 . the codec class name can be generated This is 
more efficiently handled if standardized naming conventions are used. For example from a date type of 
"video/dnepak." a suitable codec class name might be "media.codec.video.cinepakCodec." Similarty. a media date fite 
of type "audicVpcm" might be decoded/decompressed by an instance of the codec dass 
"media.codec.audio.pcm.Codec." ,„_. 
[00431 In step 402, the class name is used to perform a search for the corresponding dass using a class path vector 
as the basis for the search. The class path vector typically contains one or more path names for repositories ot hierar- 
chically distributed class files, including the currently active path to the source of the media data. A sample dass path 
might be: 

ClassPath = ./java/libAisers^smith/classes 

[0044] In order to search for class "media.codec.audio.pcm.Codec" in path "/java/lib". the directory 
"/java^ib/media/codec/audio/pcm- would be searched for the presence of a "Codec.class" file.] Each Pathname is 
searched in order until the class is found. Thus, a search of the pathname of the media server providing the media data 
win result in the successful location of a desired codec class on the server. 

[00451 In step 403, depending on whether the codec dass was found on the user computer system or on the server, 
the codec class is either loaded from persistent storage on the user computer system or downloaded fromthe server. 
The Java class loader handles the loading/downloading of the codec class byte-code file and integration of the codec 
class into the Java virtual machine. In step 404. an instance of the codec dass is returned to the player for use m 

[OMoT^ln an errtoodfmert of the invention. Ihe above method is incorporated into a codec factory object to facilitate 
the creation of plug-in codec object instances for a media application such as the JMF player. An embodiment of a 
codec factory implementation is graphically represented in the block diagram of Figure 5. . 
[00471 The embodiment of Figure 5 comprises player 500 referencing CodecFactory instance 501 . which, m turn, ref- 
erences Class instance 502. CodecFactory instance 501 contains methods getCodecClassO and createCodecO. Class 
instance 502 contains methods forNameO and newlnstenceO. TheforNameO method uses the classpath ^search 
for a dass of a specif ied class name, and returns the runtime Class descriptor for the dass wrth the specrf.ed name. 
The newlnstenceO method returns a newly allocated instance of the class represented by the Class object. The dass 
loader is invoked by newlnstenceO to load/download the given class if it is not already present in the virtual machine. 
[0048] After determining the data type of incoming media data, in step 1 . player 500 invokes the createCodecO 
method of CodecFactory 501 . specifying the codec name based on the data type. In step 2. createCodecO imvokes tte 
getc£ecC>assO method, passing the codec name from step 1 . The getCodecClassO method qua.rf.es the codec name 
with appropriate package prefixes and suffixes as needed, and in step 3, invokes the forNameO method of Class 

40 instance 502 using the qualified dass name. 

[0049] In step 4, the forNameO method uses the ClassPath vector 503 to search server site 504 and user srte 505 
represented by arrows 4a and 4b. respectively) to locate the specified codec class. In step 5. the forNameO m^od of 
Class instance 502 returns the Class descriptor for the codec class to the getCodecClassO method of CodecFactory 
instance 501. In step 6. the Class descriptor is returned to the createCodecO method. 

[0050] In step 7 the createCodecO method invokes the newlnstenceO method on the Class descriptor to create an 
instance of the represented codec dass. In step 8a (or 8b). the newlnstenceO method uses the dass loader to access 
server site 504 (or user site 505) to facilitate downloading (or loading) of the codec dass into the virtual rrachine in step 
9a (or 9b) In step 10. the newlnstenceO method returns the codec dass instance to Ihe createCodecO method of 
CodecFactory instance 501 . which then returns the codec dass instance to player 500 in step 1 1 . 

Fmhndiment of Computer Execution Environment (Hardware ! 

[0051] An embodiment of the invention can be implemented as computer software in the form of computer readable 
program code executed on a general purpose computer such as computer 600 illustrated in Figure 6 or in the form of 
byte-code dass f iles executable by a virtual machine running on such a computer. A keyboard 610 and mouse 61 1 are 
coupled to a bi-directional system bus 618. The keyboard and mouse are for introducing user input to the computer sys- 
tem and communicating that user input to central processing unit (CPU) 613. Other suitable input devices may be used 
in addition to. or in place of. the mouse 61 1 and keyboard 610. I/O (input/output) unit 619 coupled to bi-directional sys- 
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tern bus 618 represents such I/O elements as a printer. A/V (audio/video) I/O. etc. 

[0052] Computer 600 includes a video memory 614. main memory 61 5 and mass storage 612. all coupled to bi^irec- 
Sona. system bus 618 along with keyboard 610. mouse 611 and CPU 613. The mass storage 
fixed and removable media, such as magnetic, optical or magnetic opt.cal storage systems or any other available .mass 
storaae technology. Bus 618 may contain, fa example, thirty-two address lines for addressing video memory 614 or 
2 S 6?1 The system bus 61 8 also includes, for example, a 32-bit data bus for transfernng data between and 
a^wg ^components, such as CPU 613. main memory 615. video memory 614 and mass storage 612. Alternatively, 
multiplex data/address lines may be used instead of separate data and address lines. Motornla ® such as 

[00531 In one embodiment of the invention, the CPU 613 is a microprocessor manufactured by Motorola . sucn as 
he^OXO processor or a microprocessor manufactured by Intel®, such as the 80X86. or Pentium® P^or,°ra 
SPARC® microprocessor from Sun Microsystems®. However, any other suitable ^^[^J^^^ 
may be utilized. Main memory 615 is comprised of dynamic random access memory (DRAM). VWeo memory £14 .s a 
dual-ported video random access memory. One port of the video memory 614 is coupled to video amplrfie r^The 
videolmplrfier 616 is used to drive the cathode ray tube (CRT) raster monitor 617. Vkteo amplifier 616 ,s weUl^wmn 
the art and may be implemented by any suitable apparatus. This circuitry converts pixel data stored ,n vrdeo memory 
614 to a rasS signa^uitable for use by monitor 617. Monitor 617 is a type of monitor suitable for disp.ay.ng graph,c 

1ST Computer 600 may also include a communication interface 620 coupled to bus 618. Communication interface 
eXovSrato^ay ^communication coupling via a network link 621 to a local network 622. 
communication interface 620 is an integrated services digital network (ISDN) card or a modem, ^mun^oninte^ 
face 620 provides a data communication connection to the corresponding type of telephone l.ne. which comprises part 
of network link 621. If communication interlace 620 is a local area network (LAN) card, commumcaton interface , 620 
proves a data communication connection via network .ink 621 to a compatible LAN. Wreless links are a'soposs,b.e. 
Yn any such implementation, communication interface 620 sends and receives electncal. electromagnets or optical s.g- 
25 nals which carry digital data streams representing various types of '"formation. riwices 
[0055] Network link 621 typically provides data communication through one or more networks to ^er data devices- 
FonSampte. network link £>1 may provide a connection through .oca. network 622 to host cornputer 623 or to date 
equipmerJ operated by an Internet Service Provider (ISP) 624. ISP 624 in turn f^^^S^lSIZ 
torough the world wide packet data communication network now commonly referred to as Ihe | rt £ n J *2S Lcx*l ^ 
work622 and Internet 625 both use electrical, electromagnetic or optical srgnals which carry d.grra date *ear«. The 
signals through the various networks and the signals on network link 621 and through «>"imuni^to^ "^^l 
which carry the digital data to and from computer 600. are exemplary forms of earner waves transporting the intorma- 

ISuSe] Computer 600 can send messages and receive data, including program code, through the "^^^k 
Enk621 . andc^mmunication interface 620. In the Internet example, server 626 m,gr^ransm,t a requested code for an 
apportion program through Internet 625. ISP 624. local network 622 and communication interfaca ^20. Ina^rt I wrth 
invention, one such downloaded application is the apparatus for proving software plug-m components described 

SSSl The received code may be executed by CPU 613 as it is received, and/or stored in mass storage £12. or other 
non-volatile storage for later execution. In this manner, computer 600 may obtain application code .n the form of a car- 

S Ve The computer systems described above are for purposes of example only. An embodiment of the invention 
mav be implemented in any type of computer system or programming or processing environment 
maybe ^ men ^ hod ^XparatTfor providing plug-in media decoders has been described ,n conjunct™ wrth 
one or more specific embodiments. The invention is defined by the claims and their full scope of equivalents. 
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Class Hierarchy 

class java.lang.Object 

interface corn.sun.media.Codec 
interface com.sun.media.CodecOutput 
class com.sun.media.Data 
interface com.sun.media.Format 

class com.sun.media.MediaCodec (implements com.sim.media.Codec) 

class com.sun.media.MediaFrameCodec 
class com.sun.media.MediaFormat (implements com.sun.media.Format) 



Interface com.sun.media.Codec 
public interface Codec 

The Codec interface declares a set of methods that any codec will need to 
implement in order to convert encoded/compressed audio or video data to a 
format that can be understood by a rendering node or device. 



[Method Index] 

• checkFormat(Data) 

This method is invoked by the framework before decode() is called to 
verify that the input data matches the current expected input format. 

• decode(Data) 

This method does the actual data conversion. 

• getlnputFormatQ 

Returns the format of the input data that the codec is expecting. 

• getMaxScale() 

Returns the maximum scaling factor the codec can support. 

• getMinScale() 

Returns the minimum scaling factor the codec can support. 

• eetOutputFormat() 

Returns the format of the output produced by the codec after a call to 

decode(...). 

• getScaleO 

Returns the current scaling factor. 

• initialize(CodecOutput) 
Initialize the codec. 

• setScale(float) 
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Set the current scaling factor. 
[Methods] 
• initialize 

public abstract boolean initialize(CodecOutput output) 
Initialize the codec. "Connect" the codec to its output. 



Parameters: 

output - output of the codec. 

75 Returns: 

true if it's successful. 

• decode 

public abstract boolean decode(Data inData) 

This method does the actual data conversion. If the conversion failed 
for any reason, it returns false. 

Parameters: 

inData - input data. 
Returns: 

true if decoding is successful. 

• checkFormat 

pubUc abstract boolean checkFormat(Data inData) 

This method is invoked by the framework before decode() is called to 
verify that the input data matches the current expected input format. 
For codecs that can deal with changing formats, this is the place the 
codec is allowed to have a glimpse of the inpt data and change the 
decoding format before actually decoding the data in decodeQ. 



Parameters: 

inData - the input data. 

45 Returns: , . , , - - 

true if the given data is of the expected input format; false it 
the given data does not have the right input format. 



• getlnputFormat 

public abstract Format getlnputFormatQ 
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Returns the format of the input data that the codec is expecting. 

See Also: 
decode 

• getOutputFormat 

public abstract Format getOutputFormat() 

Returns the format of the output produced by the codec after a call 
decode(...). 

See Also: 
decode 

• getMinScale 

public abstract float getMinScale() 

Returns the minimum scaling factor the codec can support. 

Returns: 

the (linear) scaling factor. 

• getMaxScale 

public abstract float getMaxScale() 

Returns the maximum scaling factor the codec can support. 

Returns: 

the (linear) scaling factor. 

• getScale 

public abstract float getScale() 

Returns the current scaling factor. 

Returns: 

the (linear) scaling factor. 

• setScale 

public abstract float setScale(float scale) 

Set the current scaling factor. 

Returns: 

the (lineas) scaling factor set. 
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Interface com.sun.media.CodecOutput 
public interface CodecOutput 

An abstraction of the output interface from a codec. 



[Method Index] 

• decodeFailureUpdate(Data) 

Gets called when the decoding of a "frame" fails. 

• decodeUpdate(Data) 

Gets called when a "frame" is done decoded. 

• getOutputBuffer(Format) 

Gets an output buffer to hold the decoded bits. 

• tryGetOutputBuffer(Format) 

A non-blocking version of getOutputBuffer. 

[Methods] 

• getOutputBuffer 

public abstract Data getOutputBuffer(Format fmt) 

Gets an output buffer to hold the decoded bits. It blocks until the 
requested output buffer is available. 

Parameters: 

fmt - specifies the format of the buffer requested. 
Returns: 
the buffer. 

• tryGetOutputBuffer 

public abstract Data tryGetOutputBuffer(Format fmt) 
A non-blocking version of getOutputBuffer. 
Parameters: 

fmt - specifies the format of the buffer requested. 
Returns: 
the buffer. 

• decodeUpdate 
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public abstract void decodeUpdate(Data decoded) 

Gets called when a "frame" is done decoded. 

Parameters: 

decoded - the decoded "frame". 

• decodeFailureUpdate 

public abstract void decodeFailureUpdate(Data decoded) 
Gets called when the decoding of a "frame" fails. 
Parameters: 

decoded - the buffer to be returned back to the pool of 
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Class com.sun.media.Data 
java.lang.Object 

+ com.sun.media.Data 



public abstract class Data 
extends Object 

Base class for audio/video data. 



[Variable Index] 

• duration 

• EOM 

getLength() == EOM signifies the last chunk of data from the media. 

• presentationTime 

[Constructor Index] 

• DataO 
[Method Index] 

• getBuffer() 

Retrieve the internal buffer. 

• getBufferSize() 

• getDiscard() 

Indicate whether this data should be discarded or not 

• getDurationQ 

• getFormat() 

• getLength() 

• getPresentationTimeO 

• setDiscard(boolean) 

Set whether this data should be discarded or not. 

• setDuration(long) 

Set the duration of the data. 

• setFormat(Format) 
set data format. 

• setLength(int) 

Mark the length of data stored in the byte array. 
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• setPresentationTime(long) 

Set the presentation time of this particular segment of media data. 

[Variables] 

• presentationTime 
protected long presentationTime 

• duration 

protected long duration 

• EOM 

public static final int EOM 

getLength() == EOM signifies the last chunk of data from the media. 
[Constructors] 

• Data 
public Data() 
[Methods] 

• getFormat 

public abstract Format getFormatQ 

Returns: 

the data format. 

• setFormat 

public abstract boolean setFormat(Format f) 
set data format. 

• getBuffer 

public abstract Object getBuffer() 

Retrieve the internal buffer. An Object is returned. I could be of 
byte[] or int[]. "instanceof 1 can be used to find out. 
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Returns: 

the buffer object. 

• getBufferSize 

public abstract int getBufferSize() 

Returns: 

the size of the buffer. 

• getLength 

public abstract int getLength() 
Returns: 

the length of data stored in the byte array. EOM if it's the last 
chunk of data from the media. 

• setLength 

public abstract void setLength(int 1) 

Mark the length of data stored in the byte array. 

Parameters: 

1 - length of the data. 

• getPresentationTime 

public long getPresentationTime() 
Returns: 

the presentation time of this particular segment of media data. 

• setPresentationTime 

public void setPresentationTime(long t) 

Set the presentation time of this particular segment of media data. 

Parameters: 

t - the presentation time. 

• getDuration 

public long getDuration() 
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Returns: 

the duration of this data. 

• setDuration 

public void setDuration(long duration) 

Set the duration of the data. 

Parameters: 

duration - the media duration. 

• setOiscard 

public void setDiscard(boolean f) 

Set whether this data should be discarded or not. 

• getDiscard 

public boolean getDiscardQ 

Indicate whether this data should be discarded or not. 
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Interface com.sun.media.Format 

5 

public abstract interface Format 

Operations to allow format negotiation. It is common to describe media data 
10 as a collection of attributes that can take values from a prescribed set. 

For example audio data often has a sample rate expressed as a floating point 
number. It is also common to describe media devices and media processing as 
capable of operating with media data over a broad range of possible values 
is for particular attributes. Thus an audio renderer may be capable of 

accepting audio data with any sample rate between the ranges of 8000 and 
48000. The format interface provides methods to facilitate the negotiation 
between objects that describe media data. 

so 



[Method Index] 
s • match(Format) 

Return a Format compatible with the argument. 
30 [Methods] 

• match 

as public abstract Format rrtatch(Format other) 

Return a Format compatible with the argument. If this format cannot 
find a format compatible object null is returned. 

40 

Note: match is not commutative. 

Parameters: 

other - The format to attempt to match to. 

45 
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Class com.sun.media.MediaCodec 

java.lang.Object 
I 

+ — com.sun.media.MediaCodec 



public abstract class MediaCodec 
extends Object 
implements Codec 

This is an implementation base class of the Codec interface to convert 
encoded /compressed audio or video data to a format that can be understood 
by 

a rendering node or device. 



[Constructor Index] 

• MediaCodec() 

• MediaCodec(CodecOutput) 

[Method Index] 

• checkFormat(Data) 

This method is invoked by the framework before decode() is called to 
verify that the input data matches the current expected input format. 

• decode(Data) 

Decode the incoming bits. 

• decodeFailureUpdate(Data) 

Call this when the decoding of a "frame" fails. 

• decodeUpdate(Data) 

Call this when a "frame" is done decoding to put the decoded "frame" to 
the rendering node. 

• getInputFormat() 

Returns the format of the input data that the codec is expecting. 

• getMaxScale() 

Returns the maximum scaling factor the codec can support. 

• getMinScale() 

Returns the minimum scaling factor the codec can support. 

• getOutputBuffer(Format) 

Get an output buffer from the downstream node. 

• getOutputFormat() 
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Returns the format of the output produced by the codec after a call to 
decode(...). 

• getScale() 

Returns the current scaling factor. 

• initialize(CodecOutput) 

• setScale(float) 

Set the current scaling factor. 

• tiyGetOutputBuffer(Format) 

A non-blocking version of getOutputBuffer. 

[Constructors] 

• MediaCodec 
public MediaCodecO 

• MediaCodec 

public MediaCodec(CodecOutput o) 
[Methods] 

• initialize 

public boolean initialize(CodecOutput n) 

• decode 

public abstract boolean decode(Data inData) 

Decode the incoming bits. Call notifyFrameQ when a "frame" is 
completely decoded. 



Parameters: 

inData - input compressed data. 
40 Returns: 

whether the decoding is successful or not. 



► checkFormat 

public boolean checkFormat(Data inData) 

This method is invoked by the framework before decodeO is called to 
verify that the input data matches the current expected input format. 
For codecs that can deal with changing formats, this is the place the 
codec is allowed to have a glimpse of the inpt data and change the 
decoding format before actually decoding the data in decoded out). 
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Parameters: 

inData - the input data. 
Returns: 

true if the given data is of the expected input format. 

• getlnputFormat 

public abstract Format getInputFormat() 

Returns the format of the input data that the codec is expecting. 

See Also: 
decode 

• getOutputFormat 

public abstract Format getOutputFormat() 

Returns the format of the output produced by the codec after a call 
decode(...). 

See Also: 
decode 

• getMinScale 

public float getMinScaleQ 

Returns the minimum scaling factor the codec can support. 

Returns: 

the (linear) scaling factor. 

• getMaxScale 

public float getMaxScale() 

Returns the maximum scaling factor the codec can support. 

Returns: 

the (linear) scaling factor. 

• getScale 

public float getScaleQ 
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Returns the current scaling factor. 

Returns: 

the (linear) scaling factor. 

• setScale 

public float setScale(float scale) 

Set the current scaling factor. 

Returns: 

the (lineas) scaling factor set. 

• getOutputBuffer 

protected Data getOutputBuffer(Format fmt) 

Get an output buffer from the downstream node. This is a utility method 
for use by subclasses of MediaCodec for the actual codec 
implementation. 

Parameters: 

fmt - specifies the format of the buffer requested. 
Returns: 
the buffer. 

• tryGetOutputBuffer 

protected Data tryGetOutputBuffer(Format fmt) 

A non-blocking version of getOutputBuffer. This is a utility method for 
use by subclasses of MediaCodec for the actual codec implementation. 

Parameters: 

fmt - specifies the format of the buffer requested. 
Returns: 
the buffer. 

• decodeUpdate 

protected void decodeUpdate(Data decoded) 

Call this when a ,, frame ,, is done decoding to put the decoded "frame" to 
the rendering node. This is a utility method for use by subclasses of 
MediaCodec for the actual codec implementation. 
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Parameters: 

decoded - the decoded "frame". 

• decodeFailureUpdate 

protected void decodeFailureUpdate(Data decoded) 

Call this when the decoding of a "frame" fails. This is a utility 
method for use by subclasses of MediaCodec for the actual codec 
implementation. 

Parameters: 

decoded - the buffer to be returned back to the Tenderer. 
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Class com.sun.media.MediaFrameCodec 

java.lang.Object 
I 

+ — com.sun.media.MediaCodec 
I 

+ com.sun.media. MediaFrameCodec 



public abstract class MediaFrameCodec 
extends MediaCodec 



An implementation class to decode a "frame" of compressed data to a 
"frame" 

of decompressed data. A "frame" doesn't have to refer to a "video frame. 
It's just a logical unit of data. 



[Constructor Index] 

• MediaFrameCodec() 

• MediaFrameCodec(CodecOutput) 

[Method Index] 

• allowDropFrame(boolean) 

Switch on or off the mechanism to drop frame. 

• decode(Data) , , . , . . 

Gets called by the framework with a chunk of compressed data. 

• decode(Data, Data) 

Implement this to do the real decoding work. 

[Constructors] 

• MediaFrameCodec 
public MediaFrameCodec() 

• MediaFrameCodec 

public MediaFrameCodec(CodecOutput o) 
[Methods] 
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• decode 

public abstract boolean decode(Data in, Data out) 
Implement this to do the real decoding work. 

Parameters: 

in - input compressed data. 

out - output compressed data. 
Returns: 

true if the decoding is successful. 

• allowDropFrame 

public void allowDropFrame(boolean on) 

Switch on or off the mechanism to drop frame. If drop frame is on, the 
system will decide to drop frame if the frame is already too late to be 
presented. In such case, the frame will not reach the decode() 
function. Whether to take advantage of this is up to the codec writer. 
The advantage is that for compute intensive codecs, a delayed frame 
will not need to be processed if it will not be presented anyway. That 
saves quite some CPU cycles. This will not work for codec's that 
couldn't handle drop frames well. 

Parameters: 

on - turn on/off the frame dropping code. 

• decode 

public boolean decode(Data inData) 

Gets called by the framework with a chunk of compressed data. 

Parameters: 

inData - input compressed data. 
Returns: 

true if the decoding is successful. 
Overrides: 

decode in class MediaCodec 
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Class com.sun.media.MediaFormat 

java.lang.Object 
I 

+ com.sun.media.MediaFormat 



public abstract class MediaFormat 
extends Object 
implements Format 



[Constructor Index] 

• MediaFormat() 
[Method Index] 

• clone() 

• getCodec() 

• getDuration() 

• match(Format) 

• normalizeCodecName(String) 

[Constructors] 

• MediaFormat 
public MediaFormat() 
[Methods] 

• match 

public Format match(Format other) 

• clone 

public abstract Object clone() 

Overrides: 

clone in class Object 
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• getDuration 

public abstract long getDuration() 

• getCodec 

public abstract String getCodecO 

• normalizeCodecName 

public static String normalizeCodecName(String codec) 



20 

[0060] The features disclosed in the foregoing description, in the claims and/or in the accompanying drawings may, 
both separately and in any combination thereof, be material for realising the invention in diverse forms thereof. 

Claims 

25 

1 . A method of providing a plug-in decoder comprising: 

downloading data from a source location on a network, said data to be processed by an application; 
determining a data type of said data [to be processed by an application]; 
30 determining a plug-in decoder class associated with said data type; 

searching a persistent storage at said source location for said plug-in decoder class; and 
loading said plug-in decoder class from said persistent storage into said application. 

The method of claim 1 , wherein said step of determining said data type comprises the step of parsing header infor- 
mation associated with said data. 

The method of claim 1. wherein said step of determining said plug-in decoder class comprises the step of deter- 
mining a decoder class name from said data type. 

The method of claim 1 , wherein said step of searching said persistent storage comprises the steps of: 

determining a path name of said source location from a class path vector; and 
searching a directory in said persistent storage based on said path name. 

The method of claim 1 , wherein said steps of searching said persistent storage and loading said plug-in decoder 
class are implemented within at least one method of an instance of a factory class, said at least one method being 
invoked by said application. 



6. An apparatus for providing plug-in components comprising: 

so 

an application running on a first computer; 

persistent storage on a second computer, said persistent storage comprising data for said application and a 
plug-in component associated with a data type of said data; 

computer code executing on said first computer, said computer code configured to load said plug-in compo- 
55 nent from said second computer into said application. 

7. The apparatus of claim 6, wherein said plug-in component is a codec. 
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8, The apparatus of claim 6, wherein said application is configured to determine said data type from said data. 

9. The apparatus of claim 8, wherein said application is configured to determine a name of said plug-in component 
from said data type. 

5 10. The apparatus of claim 9, wherein said computer code is configured to search said persistent storage for said plug- 
in component based on said name provided by said application. 

11. The apparatus of claim 10. wherein said computer code is configured to perform said search using a class path 
10 vector comprising a path name of a directory in said second computer. 

12. A computer program product comprising: 

a computer usable medium having computer readable program code embodied therein for providing a plug-in 
is component, said computer program product comprising: 

computer readable program code configured to cause a computer to download data from a source location on 
a network to be processed by an application; 

computer readable program code configured to cause a computer to determine a data type of said data; 
computer readable program code configured to cause a computer to determine a plug-in component associ- 
20 ated with said data type; 

computer readable program code configured to cause a computer to search for said plug-in component, said 
search comprising a search of a persistent storage at said source location; and 

computer readable pro-am code configured to cause a computer to load said plug-in component into said 
application. 

25 , 

13. The computer program product of claim 12. wherein said plug-in component is a codec. 

14 The computer program product of claim 12. wherein said computer readable program code configured to cause a 
computer to determine said data type comprises computer readable program code configured to cause a computer 

30 to parse header information associated with said data. 

15 The computer program product of claim 12. wherein said computer readable program code configured to cause a 
computer to determine said plug-in component comprises computer readable program code configured to cause a 
computer to determine a plug-in component name from said data type. 

35 1 6. The computer program product of claim 1 2, wherein said computer readable program code configured to cause a 
computer to search for said plug-in component further comprises: 

computer readable program code configured to cause a computer to determine a path name of said source 
40 location from a class path vector; and . 

computer readable program code configured to cause a computer to search a directory in said persistent stor- 
age at said source location based on said path name. 

17. A computer data signal embodied in a carrier wave and representing sequences of instructions which, when exe- 
cuted by a processor, cause said processor to provide plug-in components by performing the steps of: 

downloading data from a source location on a network to be processed by an application; 
determining a data type of said data; 

determining a plug-in component associated with said data type; ' 
searching for said plug-in component, said search comprising a search of a persistent storage at said source 
location; and 

loading said plug-in component into said application. 

18. The computer data signal of claim 1 7, wherein said plug-in component is a codec. 

19. The computer data signal of claim 17, wherein said step of determining said data type comprises the step of pars- 
ing header information associated with said data. 
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The computer data signal of claim 17, wherein step of determining said plug-in component comprises the step of 
determining a plug-in component name from said data type. 

The computer data signal of claim 1 7, wherein said step of searching for said plug-in component further comprises: 

determining a path name of said source location from a class path vector; and 
searching a directory in said persistent storage at said source location based on said path name. 
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